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Abstract — Channel uncertainty and co-channel interference 
are two major challenges in the design of wireless systems 
such as future generation cellular networks. This paper studies 
receiver design for a wireless channel model with both time- 
varying Rayleigh fading and strong co-channel interference of 
similar form as the desired signal. It is assumed that the channel 
coefficients of the desired signal can be estimated through the use 
of pilots, whereas no pilot for the interference signal is available, 
as is the case in many practical wireless systems. Because the 
interference process is non-Gaussian, treating it as Gaussian noise 
generally often leads to unacceptable performance. In order to 
exploit the statistics of the interference and correlated fading in 
time, an iterative message-passing architecture is proposed for 
joint channel estimation, interference mitigation and decoding. 
Each message takes the form of a mixture of Gaussian densities 
where the number of components is limited so that the overall 
complexity of the receiver is constant per symbol regardless 
of the frame and code lengths. Simulation of both coded and 
uncoded systems shows that the receiver performs significantly 
better than conventional receivers with linear channel estimation, 
and is robust with respect to mismatch in the assumed fading 
model. 



I. Introduction 

With sufficient signal-to-noise ratio, the performance of 
a wireless terminal is fundamentally limited by two major 
factors, namely, interference from other terminals in the sys- 
tem and uncertainty about channel variations [1]. Although 
each of these two impairments has been studied in depth 
assuming the absence of the other, much less is understood 
when both are significant. This work considers the detection 
of one digital signal in the presence of correlated fading and 
an interfering signal of the same modulation type, possibly 
of similar strength, and also subject to independent time- 
correlated fading. Moreover, it is assumed that the channel 
condition of the desired user can be measured using known 
pilots interleaved with data symbols, whereas no pilot from the 
interferer is available at the receiver. The practical motivation 
for this problem is the orthogonal frequency-division multiple 
access (OFDMA) downlink with a single dominant co-channel 
interferer in an adjacent cell, which is typical in fourth 
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generation cellular networks. Such a situation also arises, for 
example, in peer-to-peer wireless networks. 

This work focuses on a narrowband system with binary 
phase shift keying (BPSK) modulation, where the fading 
channels of the desired user and the interferer are modeled 
as independent Gauss-Markov processes.' A single transmit 
antenna and multiple receive antennas are assumed first to de- 
velop the receiver, while extensions to more elaborate models 
are also discussed. 

The unique challenge posed by the model considered is the 
simultaneous uncertainty associated with the interference and 
fading channels. A conventional approach is to first measure 
the channel state (with or without interference), and then 
mitigate the interference assuming the channel estimate is 
exact. Such separation of channel estimation and detection 
is viable in the current problem if known pilots are also 
embedded in the interference. As was shown in [2], knowledge 
of pilots in the interfering signal can be indispensable to 
the success of linear channel estimation, even with iterative 
Turbo processing. Without such knowledge, linear channel 
estimators, which treat the interference as white Gaussian 
noise, provide inaccurate channel estimates and unacceptable 
error probability in case of moderate to strong interference. 

Evidently, an alternative approach for joint channel estima- 
tion and interference mitigation is needed. In the absence of 
interfering pilots, the key is to exploit knowledge of the non- 
Gaussian statistics of the interference. The problem is basi- 
cally a compound hypothesis testing problem (averaged over 
channel uncertainty). Unfortunately, the Maximum Likelihood 
(ML) detector becomes computationally impractical since it 
must search over (possibly a continuum of) combined channel 
and interference states for all interferers. 

In this paper, we develop an iterative message-passing algo- 
rithm for joint channel estimation and interference mitigation, 
which can also easily incorporate iterative decoding of error- 
control codes. The algorithm is based on belief propagation 
(BP), which performs statistical inference on graphical models 
by propagating locally computed "beliefs" [3]. BP has been 
successfully applied to the decoding of low-density parity- 
check (LDPC) codes [4], [5]. Other related applications of 
BP include combined channel estimation and detection for 
a single-user fading channel or frequency selective channel 
[6]-[9], multiuser detection for CDMA with ideal (nonfading) 
channels based on a factor graph approach [10] (see also 
[11], [12]), and the mitigation of multiplicative phase noise 

'The desired user and the interferer are modeled as independent. In 
principle, the fading statistics can be estimated and are not needed a priori. 
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in addition to thermal noise [13]-[15]. Unique to this paper is 
the consideration of fading as well as the presence of a strong 
interferer. This poses additional challenges, since the desired 
signal has both phase and amplitude ambiguities, which are 
combined with the uncertainty associated with the interference. 
The following are the main contributions of this paper: 

1) A factor graph is constructed to describe the model, 
based on which a BP algorithm is developed. For a finite 
block of channel uses, the algorithm performs optimal 
detection and estimation in two passes, one forward and 
one backward. 

2) For practical implementation, the belief messages (con- 
tinuous densities) are parametrized using a small number 
of variables. The resulting suboptimal message-passing 
algorithm has constant complexity per bit (unlike the 
complexity of ML which grows exponentially with the 
block length). 

3) Decoding of channel codes of LDPC-type is also incor- 
porated in the message-passing framework. 

4) As a benchmark for performance, a lower bound for 
the optimal uncoded error probability is approximated 
by assuming a genie-aided receiver in which adjacent 
channel coefficients are revealed. 

Numerical results are presented, which show that the message- 
passing algorithm performs remarkably better than the con- 
ventional technique of linear channel estimation followed by 
detection of individual symbols with or without error-control 
coding. Furthermore, the relative gain is not substantially 
diminished in the presence of model mismatch {i.e., if the 
Markov channel model assumed by the receiver is inaccurate), 
as long as the channels do not vary too quickly. 

The remainder of this paper is organized as follows. The 
system model is formulated in Section II, and Section III de- 
velops the message-passing algorithm. A lower bound for the 
error probability is studied in Section IV. Section V discusses 
the extensions to general scenarios and the computational 
complexity of the proposed algorithm. Simulation results are 
presented in Section VI and Section VII concludes the paper. 

II. System Model 

Consider a narrow-band system with a single transmit 
antenna and Nji receive antennas, where the received signal 
at time i in a frame (or block) of length I is expressed as 

yi = hiXi + h[x\+ni i=l...l (1) 

where Xi and .t' denote the transmitted symbols of the 
desired user and interferer, respectively, hi and /i^ denote the 
corresponding AT/j-dimensional vectors of channel coefficients 
whose covariance matrices are af^I and a-f^,!, and {n^} 
represents the circularly- symmetric complex Gaussian (CSCG) 
noise at the receiver with covariance matrix a^I. For simplic- 
ity, we assume BPSK modulation, i.e., Xi,x[ are i.i.d. with 
values ±1. 

Assuming Rayleigh fading, {hi} and {h-} are modeled as 
two independent Gauss-Markov processes, that is, they are 



generated by first-order auto-regressive relations (e.g., [16]): 

hi = ahi-i + \/l — Wi (2a) 
h'i = ah'^_^ + ^Jl-a^w'^ (2b) 

where {wi} and {w[] are independent white CSCG processes 
with covariance aj^I and (t\,I, respectively, and a determines 
the correlation between successive fading coefficients. This 
model includes two special cases: a = corresponds to 
independent fading and a = \ corresponds to block fading. 
Although this model is simple, general fading model can be 
approximated by such first-order Markovian model [17], [18] 
via choosing appropriate value for a. Furthermore, numerical 
simulations in Section VI also show that the receiver designed 
under such channel assumption is robust in other fading 
environments as long as the channel variation over time is not 
too fast. Note that (1) also models an OFDM system where i 
denotes the index of sub-carriers instead of the time index. 

Typically, pilots are inserted periodically among data 
symbols. For example, 25% pilots refers to pattern 
"PDDDPDDDPDDD...", where P and D mark pilot and 
data symbols, respectively. Let yl denote the sequence 
VijVi+i, ■ ■ ,yj- The detection problem can be formulated 
as follows: Given the observations yi and known value of 
a certain subset of symbols in x{ which are pilots, we wish to 
recover the information symbols from the desired user, i.e., the 
remaining unknown symbols in x[, where the realization of the 
channel coefficients and interfering symbols is not available. 

III. Graphical Model and Message Passing 

A. Graphical Model for Uncoded System 

An important observation from (1) and (2) is that the 
fading coefficients {(^i, form a Markov chain with 
state space in C^^"". Also, given {(ft-i, ft,-)}-^^, the 3-tuple 
{xi,x'i,yi) of input and output variables is independent over 
time i = 1,2,..., I. The joint distribution of the random 
variables can be factored as 

I 

p{y[ ,x[,x'l,h[,h'l)=p{hi,h[)Jlp{hi,h[\hi-i,h'i_^) 

I 

X n (piVi\f^i'KyXi,x'i)p{Xi)p{x[)^ . 

This factorization can be described using the factor graph 
shown in Fig. 1. 

Generally, a factor graph is a bipartite graph, which consists 
of two types of nodes: the variable nodes, each denoted 
by a circle in the graph, which represents one or a few 
random variables jointly; and the factor nodes, each denoted 
by a square which represents a constraint on the variable 
nodes connected to it [3], [19]. The factor node between 
the node {hi,h^) and the node {hi-x,h\_i) represents the 
conditional distribution p{hi,h'i\hi-\,h'i_i), which is the 
probability constraint specified by (2). Similarly, the factor 
node connecting nodes y^, {hi,h'i) and {xi,x^) represents 
the conditional distribution p{yi\hi,h'i,Xi,x'i), which is the 
relation given by (1). The prior probability distribution of the 
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Fig. 1. A factor graph describing tlie communication system model 
without channel coding. The arrows refer to messages wliich are discussed in 
Section III-B 



data symbols is assigned as follows. All BPSK symbols Xi and 
x[ are uniformly distributed on { — 1,1} except for the subset 
of pilot symbols in x\, which are set to 1. The Markovian 
property of the graph is that conditioned on any cut node(s), 
the separated subsets of variables are mutually independent. 
As we shall see, the Markovian property plays an important 
role in the development of the message-passing algorithm. 

Since the graphical model in Fig. 1 fully describes the 
probability laws of the random variables given in (1) and 
(2), the detection problem is equivalent to statistical inference 
on the graph^. Simply put, we seek to answer the following 
question: Once the realization of a subset of the variables 
(received signal and pilots) on the graph is revealed, what 
can be inferred about the symbols from the desired user? 

Note that the same factor graph would arise if we were 
to jointly detect both xt and x[, which solves a problem 
of multiuser detection. In this work, however, the receiver 
is only interested in detecting Xi, so that x' is being aver- 
aged out when passing messages between nodes [hi, h[) and 

B. Exact Inference via Message Passing 

In the problem described in Section II, the goal of inference 
is to obtain or approximate the marginal posterior probabiUty 
p{xi\y\), which is in fact a sufficient statistic of y\ for Xi. 
Problems of such nature have been widely studied (see e.g., 
[22, Chapter 4] and [3]). In particular, BP is an efficient 
algorithm for computing the posteriors by passing messages 
among neighboring nodes on the graph. In principle, the result 
of message passing with sufficient number of steps gives 
the exact a posteriori probability of each unknown random 
variable if the factor graph is a tree (i.e., free of cycles). For 
general graphs with few short cycles, iterative message-passing 
algorithms often compute good approximations of the desired 
probabilities. Unlike in most other work (including [13]-[15]), 
where each random variable is made a variable node, multiple 

-The style of the factor graph in Fig. 1 is similar to that used in some other 
work, such as [7], [13] and [20], which address channel and phase uncertainty 
in the absence of interference. Note that there are several different styles of 
factor graphs, e.g., the Forney style [21] which uses edges rather than circular 
nodes to represent variables. 



variables are clustered into a single node so that the factor 
graph in Fig. 1 is free of cycles (see also [3] for the usage 
of the clustering techniques). Numerical experiments (omitted 
here) show that making each variable a separate node leads to 
poor performance due to a large number of short cycles {e.g., 
there will be a cycle through hi, hi, /li+i, h'i_^_i). 

Let Gi = [hi,h'i] and Ui = [wi,w'^. The model (1) and 
(2) can be rewritten as: 



G, = aG,-i + Vl - 



(3) 
(4) 



The probability distributions immediately available are 

p{y^\Gi,Xi,x'^, p{Gi\Gi^i) and the marginals p{xi), 
p{x'^), as well as p{Gi) which are Gaussian. Note that 
p{yi\Gi,Xi,Xi) is the conditional Gaussian density corre- 
sponding to the channel model (1) and p{xi, x'i) — p{xi)p{x[) 
since the desired symbol and the interference symbol are 
independent. Also, P{xi = 1) = 1 and P{xi ~ —1) = if a;i 
is a pilot for the desired user, otherwise P(xi = ±1) = 1/2. 
Moreover, P{x'i = ±1) = 1/2 for all i, since we do not know 
the pilot pattern of the interfering user 

The goal is to compute for each i = 1, . . . ,1: 

P{xi\y[)= J Pi^^^^'z^G^\y[)dGi 

X^ — ±1 

« ^ J p{x^,x'^,y\~^,y^,y\_^^l,G^)dG^ 

where the "proportion" notation oc indicates that the two sides 
differ by a factor which depends only on the observation 
y\ (which has no influence on the likelihood ratio P{xi = 
l\y{)/ P{xi = ~l\y{) and hence on the decision). For 
notational simplicity we have also omitted the limits of the 
integrals, which are over the entire axes of 2Nfi complex 
dimensions. By the Markovian property, (x^, x^, yj, y]^^ and 
y\_^_l are mutually independent given Gi. Therefore, 



p{x,\y[) cx 



p{yt,Xt,x[\Gi)p{y\ ^\G^) 

xp{y\^,\GMG'ddG, 



x'.=±l 



X 



piGM+i)/p{Gi)dG, (5) 



where the independence of {xi,Xi) and Gi is used to ob- 
tain (5). In order to compute (5), it suffices to compute 

piG,,\y\-') and p{GU+i)- 

We briefly derive the posterior probability p{Gi\y\ ^) as 
a recursion in below, whereas computation of p{Gi\y\_^_i) is 
similar by symmetry. Consider the posterior of the coefficients 
Gi given the received signal up to time i — 1. The influence 
of y]~^ on Gi is through Gi-i because Gi and y\^^ are 
independent given Gi-i. Thus, 



p{G^\y 



G,|G,_iMG^-i|yr')dG,_i 
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By the Markovian property, yl'"^ and y^.^ are independent Algorithm 1 Pseudo code for the message-passing algorithm 



given Gi-i, so that 

p(G,|yri)a/p(G,|G,_i)p(yi-i|Gi-iMGi-i|yi-')dGi_i. 

(6) 

Since Gi-i and Xi-i,x[_i are independent, 

p{yi_i\G^-i) 

= X! Myi-iki-i,a;i-i,Gi_i)p(xi_i)p(a;-_i). (7) 
a;«-i,x<_i=±l 

Therefore, by (6) and (7), we have a recursion for computing 
p{Gi\y]~^) for each i = 1, . . . ,1, 



-V E / 

Xi_i,a;^_^=±l 



p(Gi|Gi_iMGi_i|yr2) 



X p(yj_i|Gi_i,a;i_i,a;-_i)p(a;i_i)p(a;-_i)dGi_i (8) 

which is the key to the message-passing algorithm. Similarly, 
we can also derive the inference on Gi, which serves as an 
estimate of channel coefficients at time i: 

p{Gi\y[)(x E P{3:i)p{x'i)p{yi\Gi,Xi,x'i) 

xp{Gi\y{-')p{Gi\yl+,)/p{Gi). (9) 

In other words, the BP algorithm requires backward and 
forward message-passing only once in each direction, which 
is similar to the BCJR algorithm [23]. The key difference 
between our algorithm and the BCJR is that the Markov chain 
here has a continuous stale space. ^ 

The joint channel estimation and interference mitigation 
algorithm is summarized in Algorithm 1. Basically, the mes- 
sage from a factor node to a variable node is a summary of 
the extrinsic information"* (EI) about the random variable(s) 
represented by the variable node based on all observations 
connected directly or indirectly to the factor node [3]. For 
example, the message received by node {hi, h'^) from the factor 
node on its left summarizes all information about {hi,h[) 
based on the observations t/j^, . . . , which is proportional 
to p{hi,h\\yY'^). The message from a variable node to a 
factor node is a summary of the EI about the variable node 
based on the observations connected to it. For example, the 
message passed by node {hi, h'^) to the factor node on its left 
is the EI about {hi, h[) based on the observations y^,. . . ,y^, 
i.e., p{hi,h'i\y\). 

Given the factor graph, it becomes straightforward to write 
out the message passing algorithm, which is equivalent to the 
sum-product algorithm [3]. Because of the simple Markovian 
structure of the factor graph, we derive the algorithm using 
basic probabihty arguments in this section. The preceding 
treatment is self-contained, and the technique also applies to 
other similar problems. 

^Another way to derive the message passing algorithm is based on the factor 
graph, in which the joint probability is factored first and then marginalized to 
get the associated posterior probability [3]. 

''It is obtain by removing the posterior probability of the variable node itself 
in the a posterior probabihty (APP). 



Initialization: P(x^ = 1) = P{Xi = -1) = 1/2 for aU i. 
The same probabilities are also assigned to p{xi) for all i 
except for the pilots, for which P{xi = 1) = 1. For all i, 
p{Gi) is zero mean Gaussian with variance Q. 
for i = 1 to Z do 

Compute p{Gi\y\-^) from (8) 

Compute p{Gi\y\_^-^) similarly to (8) 
end for 

for i = 1 to Z do 

Compute p{xi\y\) from (5) 
end for 



C. Practical Issues 

Algorithm 1 cannot be implemented directly using a digital 
computer because the messages are continuous probability 
density functions (PDFs). Here we choose to parametrize the 
PDFs, as opposed to quantizing the multi-dimensional PDFs 
directly, which requires a large number of quantization bins 
and thus high computational complexity. Also, parametrization 
can characterize the PDFs exactly without introducing extra 
quantization error. Thus it can achieve better performance with 
less complexity. (Of course, for hardware implementation the 
PDF parameters must be quantized.) 

For notational convenience, we use g to denote the column 
vector formed by stacking the columns of the matrix G, i.e.. 



if Gi 



hl,h 



/T 



hi,hi] is Nr X 2 as defined previously then = 

T 

is 2Nr X 1. We define 



, Jo = 





Xi 



X'i 





where ® denotes the Kronecker product and Ir denotes the 
r X r identity matrix. Then (3) and (4) are equivalent to 



9i = o:9t-i + ^\-o?Ui 



(10) 
(11) 



where Ui is a column vector consisting of 2iV/j indepen- 
dent CSCG variables with variance u\ or a\i. Let the 
r— dimensional complex Gaussian density be denoted by 

1 



■ cxp [— (x — m)^li' ^(a; — m)] 



where column vector of complex dimension r, and 

rrirxi and Krxr denote the mean and co variance matrix, 
respectively. Let Q = diag(c7^, a^, cr^,, cr^,). We can then 
write pigi\gi_i) = CAf{gi,agi_-^, \/l - Q), piOi) = 
CAf{g„0, Q) and p{y.^\g^, x,„x',) = CN{y„ Z^g.^.all). 

The density functions, p{gi\yi+i) and p{gi\yi ^) are Gaus- 
sian mixtures. Note that the random variables in Fig. 1 are 
either Gaussian or discrete. The forward recursion (8) for 
P{9i\y'r^) starts with a Gaussian density function. As the 
message is passed from node to node, it becomes a mixture 
of more and more Gaussian densities. Each Gaussian mixture 
is completely characterized by the amplitudes, means and 
variances of its components. Therefore, we can compute and 
pass these parameters instead of PDFs. 
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Without loss of generality, we assume that p{gi-i\y\~^) = 
PiCN{gi_i,ml_-^,Kl_-y), where non-negative numbers 
{pj} satisfy J2j Pj = 1- Substituting into (8), after some 
manipulations, we have 

P(9il2/i"')ot Y PjPixi-i)p{x'i_i)L{j,i)C{j,i) (12) 
where 

L{j,i) = CAf{Zi-imi_„y,_^,all+Zi-iK{_,Zi-i) (13) 
and 

C(j,z)=CAA(g„mf,xf) (14) 

where 

rrv>'' =ami_,+aKi_,Zt^{<ylI + Z.^^Ki_,Z,^,)-' 

X (y, - Zj_im^„J (15a) 

= c^KU + VT^Q {aK^Zt,) 

X {all + Zi_,Kl_^Zf_,r\aZi_iKl_^). (15b) 

Basically, (12), (13) and (14) give an expUcit recursive compu- 
tation for the amplitude, mean and variance of each Gaussian 
component in message p{gi\y]~^). Similar computations can 
be applied to p{g.^\yl^^). 

Examining (15a) and (15b) more closely, ignoring the 
superscripts, they are the one-step prediction equation and 
Riccati equations, respectively, for the Unear system defined 
by (3) and (4) with known [24, Ch.3], [3, Sec. IV.C]. 
Therefore, passing messages from one end to the other can be 
viewed as a series of Kahnan filters with different weights: In 
each step, each filter performs the traditional Kalman filter for 
each hypothesis of Zj_i and the filtered result is weighted by 
the product of the previous weight, the posterior probability 
of the hypothesis, and L{j,i)? 

The number of Gaussian components increases exponen- 
tially in the recursive formula (12), which becomes compu- 
tationally infeasible. In this work, we fix the total number of 
components and simply pick the components with the largest 
amplitudes (which correspond to the most likely hypotheses). 
In general, this problem is equivalent to the problem of 
survivor-reduction. Two techniques that have been proposed 
are decision feedback [26] and thresholding [27]. The former 
limits the maximum number of survivors by assuming the past 
decisions are correct, while the latter keeps the survivors only 
when their a posteriori probabiUties exceed a certain threshold 
value. According to the preceding analysis, the method we 
propose falls into the decision feedback category. Obviously, 
the more components we keep, the better performance we 
have; however, the higher the complexity at the receiver. We 
investigate this issue numerically in Section VI. A different 
approach to Umiting the number of Gaussian components 
is presented in [14], [281-[311. There the basic idea is to 
merge components "close" to each other instead of discarding 
the weakest ones as we do here. However, that requires 

'The value of L{j,i) is given by (13) and is related to the difference 
between the filtered result and the new observation. Prior work in wiiich 
a single Kalman filter is used for chaimel estimation in the absence of 
interference is presented in [20], [25]. 



computing distances between pairs of components, which can 
lead to significantly higher complexity [29], [31]. The relative 
performance of these different methods is left for future study. 

D. Integration with Channel Coding 

Channel codes based on factor graphs can be easily in- 
corporated in the message-passing framework developed thus 
far. In Fig. 2, a sparse graphical code is in conjunction with 
the factor graph for the model (1) and (2). The larger factor 
graph is no longer acyclic. Therefore, the message-passing 
algorithm is sub-optimal for this graph even if one could keep 
all detection hypotheses {i.e., the number of mixture com- 
ponents is unrestrained). However, design of channel coding 
can guarantee the degrees of such cycles are typically quite 
large. Hence message-passing performs very well [5]. Based 
on the factor graph, one can develop many message-passing 
schedules. To exploit the slow variation of the fading channel, 
the non-Gaussian property of the interfering signal and the 
structure of graphical codes, a simple idea is to allow the 
detector and decoder to exchange their extrinsic information. 
For example, suppose that at a certain message-passing stage, 
the node x^+i computes its APP from the detector. Then 
the node Xi+i can distribute the EI to the sub-graph of the 
graphical code, which is described by the solid arrows in 
Fig. 2. After Xi+i collects the "beliefs" from all its edges, 
it passes the EI (which is obtained by multiplying together 
all "behefs" but the one coming from the detector) back to 
the detector. This process is described by the dashed arrows 
in Fig. 2. In other words, both the detector and the decoder 
compute their posterior probabilities from received EI. 

In this paper, we use LDPC codes with the following simple 
strategy: We run the detection part as before and then feed 
the EI to the LDPC decoder through variable nodes Xi. After 
running the LDPC decoder several rounds, we feed back the 
EI to the detection sub-graph. We investigate the impact of 
the message-passing schedule on performance numerically in 
Section VI. 

IV. Error Floor Due to Channel Uncertainty 

Channel variations impose a fundamental hmit on the error 
performance regardless of the signal-to-noise ratio (SNR). 
Consider a genie-aided receiver: when detecting symbol Xi, 
a genie reveals all channel coefficients but {hi, h'^) to the 
receiver, which can only reduce the minimum error probability. 
Even in the absence of noise, the receiver cannot estimate 
{hi,h^) precisely (not even the sign) due to the IVlarkovian 
property in (2). Therefore, the error probability does not vanish 
as the noise power goes to zero. 

Evidently, the genie-aided receiver also gives a lower bound 
on the error probabiUty for the exact message passing algo- 
rithm. In the following, we derive an approximation to this 
lower bound. Numerical results in Section V indicate that 
the difference between the approximate lower bound and the 
actual genie-aided performance is small. 

Consider the error probabihty of jointly detecting [a;i,a;^] 
with the help of the genie. Conditioned on all other channel 
coefficients, h, and are Gaussian. Let hi = hi + hi and 
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Fig. 2. A factor graph for joint detection, estimation and decoding. Tlie solid arrows sliow EI passed from the detector to the decoder, and the dashed arrows 
show EI passed from the decoder to the detector. 



h^ = h^-\- ft,, where hi and are the estimates of hi and h^, 
respectively, and hi and hi are the respective estimation errors. 
By treating hiXi and /i,a;^ as additional noise, the channel 
model can be rewritten as 

= h^x^ + h.^x^ + fii 

where the residual noise n,; = hiXi + h^x'i + rii. It 
can be shown that fii is a CSCG random vector inde- 
pendent of {xi.x'i) and with covariance matrix (t|J — 

Let Xi and x[ be the estimates of xt and x'^, respectively. 
For simplicity, we assume dual receive antennas {Nb. — 2). 
Following a standard analysis [1, App. A], we have 

P(error) = P{xi ^ a;^,^- = a;-) + P{xi ^ Xi.x'i ^ x-) 

2 X 2 



1 - (Ul 



(2 + Aii) + 



1 - ^^2 



(2 + ^*2) (16) 



where 



Ml 



/i2 = 



v2t72 + (1 + a^)al 



Note that as long as a 7^ 1, the residual noise h does 
not vanish, which results in an error floor. Therefore, such 
error floor is inherent to the channel model, and despite its 
simplicity, the channel cannot be tracked exactly based on 
pilots. 

V. Extensions and Complexity 
A. Extensions 

The message-passing approach applies to general multiple- 
input multiple-output systems. For example, if Nt transmit 
antennas are used by the desired user, Nlj^ transmit antennas 



are used by the interferer, and Nb. antennas are used by the 
receiver, then the system can be described as 



Vi = HiXi + 
Hi — FHi^i 
H'i — F' H'i__-i 



+ W, 



rii 



(17a) 
(17b) 
(17c) 



where UiiNn x 1), Xi{NT x 1), x'-{Nfp x 1) are the received 
signal, desired user's signal and interfering signal, respectively, 
at time i, the noise ni{Njj x 1) consists of CSCG entries, and 
Hi{NR X Nt) and Hi{Nii x Nlj.) are independent channel 
matrices. Equations (17b) and (17c) represent the evolution of 
the channels, where F and F' are in general square matrices, 
and Wi and W[ are independent CSCG noises. 

Let hj i represent the j-th column of Hi, and define 

T 

./i , /T 
j, III i, iI'2Aj ■ 



, It 



N' 



T rTl 
X^ ,X^ J 



T 



T /T 



,/T 



A = E [g,gf_i 
B = E[g,;Mf ] . 



Note that (10) and (1 1) are still valid, where a and vl— Q 
are replaced by A and B, respectively. Therefore, with this 
replacement, the BP algorithm for this general model remains 
the same. 

We can also replace the Gauss-Markov model with higher 
order Markov models. By expanding the state space (denoted 
by Gi), we can still construct the corresponding factor graph 
by replacing variable nodes [Hi^H'i) with Gi, and a similar 
algorithm can be derived as before. Also, extensions to systems 
with more than one interference can be similarly derived. 

Furthermore, the proposed scheme can in principle be 
generalized to any signal constellation and any space-time 
codes, including QPSK, 8-PSK, 16-QAM and Alamouti codes. 
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SNR (dB) SNR (dB) SNR (dB) 

(a) (b) (c) 

Fig. 3. The BER performance of the message-passing algorithm. The density of pilots is 25%. (a) The power of the interference is 10 dB weaker than that 
of the desired user, (b) The power of the interference is 3 dB weaker than that of the desired user, (c) The power of the interference is identical to that of 
the desired user. 



However, as the constellation size, the space-time codebook 
size or the number of interferers increases, the complexity 
of the algorithm increases rapidly, while the advantage over 
Unear channel estimation vanishes because the interference 
becomes more Gaussian due to central limit theorem. Thus 
the algorithm proposed in this paper is particularly suitable 
for BPSK and QPSK modulations, space-time codewords with 
short block length and a small number of interferers. A 
detailed tally of the total complexity is given next. 

B. Complexity 

With or without coding, the complexity of the message- 
passing receiver is linear in the frame length, and polynomial 
in the number of antennas. 

Suppose that there are m channel coefficients, so g is a 
vector of length m (for the Gauss-Markov channel model 
TO = Nii{NT + N!p)). The number of receive antennas is 
Nn, the maximum number of Gaussian components we allow 
is C, and the sizes of the alphabet of Xi and x'^ are |^| and 
\A'\, respectively. The complexity of computing p{g^\y]^^) is 
then 0{C\A\\A'\N^l), where / is the frame length and N^^ 
is due to the matrix inverse in (15b). The exponent a depends 
on the particular inversion algorithm, and is typically between 
two and three. ^ Similar complexity is needed to compute 
p{Gi\y\_^_i). To synthesize the results from the backward and 
forward message passing via (5), we need 0(C^ |yl||yl'|TO°/) 
computations. Thus, the total complexity for the uncoded sys- 
tem is 0{{CNfi + C^TO°) 1^1 1^' 10- To reduce the complexity, 
one can reduce C, which causes performance loss. One can 
also try to approximate the matrix inverse (or equivalently, 
replace the Kalman filter with a suboptimal filter). 

For a coded system, the complexity of message-passing 
LDPC decoder is generally linear in codeword length [4]. 
With multiple frames coded into one codeword, the decoder 
complexity is also linear in the frame length. Suppose the 
number of EI exchanges between detector and decoder is Idet- 
Then the overall complexity for the receiver is 0{{CN^ + 
C^m-)Iaet\A\\A'\l). 

'The value a = 2.37 is established in [32] for general matrices. There 
has been recent progress on developing efficient algorithms for matrix 
computations [33] and the Hermite matrices in (15b) may allow a further 
reduction in complexity. 



VI. Simulation Results 

In this section, the model presented in Section II with dual 
receive antennas (N^i = 2) and BPSK signaling, is used for 
simulation. The performance of the message-passing algorithm 
is plotted versus signal-to-noise ratio SNR — <j\l<Jn, where 
the covariance matrix of the noise is cr^/. We set the channel 
correlation parameter^ a — .99 and limit the maximum 
number of Gaussian components to 8. Within each block, there 
is one pilot in every 4 symbols. For the uncoded system, we 
set the frame length to I = 200. For the coded system, we use 
a (500, 250) iiTegular LDPC code and multiplex one LDPC 
codeword into a single frame, i.e., we do not code across 
multiple frames. 

A. Performance of Uncoded System 

1) BER Performance: Results for the message-passing al- 
gorithm with the Gaussian mixture messages described in 
Section III are shown in Figs. 3 to 8. We also show the 
performance of three other receivers for comparison. The 
first is denoted by "M]VISE", which estimates the desired 
channel by taking a linear combination of adjacent received 
value. This M]VISE estimator treats the interference as white 
Gaussian noise. The second is the genie-aided receiver de- 
scribed in Section IV, denoted by "Genie-aided LB", which 
gives a lower bound on the performance of the message- 
passing algorithm. The third one is denoted by "ML with full 
CSI", which performs maximum likelihood detection for each 
symbol assuming that the realization of the fading processes 
is revealed to the detector by a genie, which lower bounds 
the performance of all other receivers. We also plot the 
approximation of the BER for the optimal genie-aided receiver 
obtained from (16) using a dashed line. 

Fig. 3 shows uncoded BER vs. SNR, where the power of the 
interference is 10 dB weaker, 3 dB weaker and equal to that of 
the desired user, respectively. The message-passing algorithm 
generally gives a significant performance gain over the MMSE 
channel estimator, especially in the high SNR region. Note 

'in Clark model, correlation between adjacent symbols is .99 corresponds 
to the scenario with the normalized maximum Doppler frequency approxi- 
mately 0.03. In the other words, o = .99 corresponds to 300 Hz of Doppler 
spread with symbol rate of 10 Kbps. 




SNR (dB) 



Fig. 4. Channel estimation eiTor with an interfere!', which is 3 dB weaker 
than the desired signal. The density of pilots is 25%. 



that thermal noise dominates when the interference is weak. 
Therefore, relatively little performance gain over the MMSE 
algorithm is observed in Fig. 3(a). In the very low SNR re- 
gion, the MMSE algorithm slightly outperforms the message- 
passing algorithm, which is probably due to the limitation on 
number of Gaussian components. 

The trend of the numerical results shows that the message- 
passing algorithm effectively mitigates or partially cancels the 
interference at all SNRs of interest, as opposed to suppressing 
it by linear filtering. We see that there is still a gap between 
the performance of the message-passing algorithm and that 
of the genie-aided receiver. The reason is that revealing the 
channel coefficients enables the receiver to detect the symbol 
of the interferer with improved accuracy. Another observation 
is that the analytical estimate is closer to the message-passing 
algorithm performance with stronger interference. 

2) Channel Estimation Performance: The channel estimate 
from the message-passing algorithm is much more accurate 
than that from the conventional linear channel estimation. 
Fig. 4 shows the mean squared error for the channel estimation 
versus SNR where the interference signal is 3 dB weaker 
than the desired signal and one pilot is used after every 
three data symbols. Note that the performance of the linear 
estimator hardly improves as the SNR increases because the 
signal-to-interference-and-noise ratio is no better than 3 dB 
regardless of the SNR. This is the underlying reason for the 
poor performance of the linear receiver shown in Fig. 3. 

B. The Impact of Imperfect Knowledge of Channel Statistics 

Although the statistical model for the channel is usually 
determined a priori, the parameters of the model are often 
based on on-line estimates, which may be inaccurate. The 
following simulations evaluate the robustness of the receiver 
when some parameters, or the model itself is not accurate. 
The simulation conditions here are the same as for the previous 
uncoded system with 3 dB weaker interference. Fig. 5 plots the 
BER performance against the correlation coefficient a, while 
the receiver uses a instead. It is clear that the mismatch in 




Fig. 5. Performance v.s. different values of a with 3 dB weaker interference. 
For curves marked with "+", the receiver uses the true value of correlation 
coefficient i.e., a = a. For curves marked with "*", receiver uses a = .99 
regardless of the value of a. 




SNR (dB) 



Fig. 6. Performance under the Clarke channel with normalized maximum 
Doppler frequency 0.02 and 3 dB weaker interference. 



a causes little degradation. The result of a similar experiment 
is plotted in Fig. 6, where the receiver assumes the Gauss- 
Markov model, while the actual channels follow the Clarke 
model [1, Ch. 2].** We see that the message-passing algorithm 
still works well. In fact, as long as the channel varies relatively 
slowly, modeling it as a Gauss-Markov process leads to good 
performance. 

C. Coded system and the Impact of Message-passing Schedule 

Consider coded transmission using a (500, 250) irregular 
LDPC code^ and with one LDPC codeword in each frame, i.e., 
no coding across multiple frames. Since we insert one pilot 
after every 3 symbols, the total frame length is 667 symbols. 

*We set a according to the auto-correlation function for the Clark model. 

'The left degree parameters are A3 = .9867, A4 = .0133; the right degree 
parameters are p4 = .0027, ps = .0565, pa = .8332, p-j = .1023, pg = 
.0053. For the meaning of the parameters, please refer to [5]. 
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Separate Message-passing A!g 

Joint Message-passing Alg. 

Linear-IUIMSEAIg. 

ML Detection witti full CSI 




SNR (dB) 

Fig. 7. The BER performance for the system with a (500, 250) irregular' 
LDPC code. The interference is 3 dB weaker than the desired signal. The 
density of pilots is 25%. "ML Detection with full CST' refers to ML detection 
with full CSI followed by BP-based LDPC decoding, which serves as a 
performance benchmark. 



For the message-passing algorithm, let /^et denote the total 
number of EI exchanges between decoder and detector, and 
Idee denote the number of iterations of the LDPC decoder 
during each EI exchange. Different values for pair {Idet,Idee) 
correspond to different message-passing schedules. 

In Fig. 7, we present the performance of two message- 
passing schedules: (a) Idet — 1 and Idee = 50 denoted by 
"Separate Message-passing Alg.", i.e., the receiver detects the 
symbol first, then passes the likelihood ratio to the LDPC 
decoder without any further EI exchanges (separate detection 
and decoding), and (b) Idet — 5 and Idee — 10, denoted by 
"Joint Message-passing Alg.", i.e., there are five EI exchanges 
and the LDPC decoder iterates 10 rounds in between each 
EI exchange. For the other two receiver algorithms, the total 
number of iterations of LDPC decoder are both 50. As shown 
in Fig. 7, the message-passing algorithm preserves a significant 
advantage over the traditional linear MMSE algorithm and the 
joint message-passing algorithm gains even more. 

The performance with different message-passing schedules 
is shown in Fig. 8. where we fix total number of LDPC 
iterations Idet x Idee- Generally speaking, if Idee or Idet x Idee 
is fixed, more EI exchanges lead to better performance. We 
also observe that when Idee is relatively large, say 30, the 
performance gain from EI exchanges is small. The reason 
is that when Idee is large, the output of the LDPC decoder 
"hardens", i.e., the decoder essentially decides what each 
information bit is. When the EI is passed to the detector, all 
symbols look like pilots from the point of view of the detector. 
Therefore, there is not much gain in this case. 

D. The Impact of Mixture Gaussian Approximation 

As previously mentioned, the number of Gaussian com- 
ponents in the messages related to the fading coefficients 
grows exponentially. For implementation, we often have to 
truncate or approximate the mixture Gaussian message. In this 




SNR (dB) 



Fig. 8. The impact of different message-passing schedules. 



paper, we keep only a fixed number of components with the 
maximum amplitudes. The maximum number of components 
clearly has some impact on the performance. Here we present 
some numerical experiments to illustrate this effect. 

When the pilot density is high, say 50%, there is no need 
to keep many Gaussian components in each message. In 
fact, keeping two components is essentially enough. However, 
when the pilot density is lower, say 25%, the situation is 
different. Fig. 9 shows the BER performance when we keep 
different numbers of Gaussian components in the message- 
passing algorithm where the pilot density is 25%. For this case, 
we need 8 components for each message passing step. Indeed, 
the lower the pilot density, the more Gaussian components we 
need to achieve the same performance. When the pilot density 
is low, we must keep a sufficient number of components, cor- 
responding to a sufficient resolution for the message. Roughly 
speaking, the number of Gaussian components needed is 
closely related to the number of hypotheses arising from 
symbols between the symbol of interest and the nearest pilot. 

For a single-user system, previous studies indicate that a 
single Gaussian approximation of the messages is sufficient, 
e.g., [20]. Fig. 9 shows that this is not the case for the 
system considered with one dominant interferes For the sim- 
ulation with the single Gaussian approximation all Gaussian 
components are combined into one at each message-passing 
stage according to the minimum divergence criterion [31]. 
As expected, the performance with the single Gaussian ap- 
proximation is relatively poor. Namely, consider the posterior 
probability, or conditional PDF, of {hi,h'^), which is the 
message passed along the graph. Due to the lack of pilots for 
estimating h'^, there is inherent ambiguity of its polarity so 
that the posterior of h'^ is always symmetric around the origin 
even with an exact message-passing algorithm. Consequently, 
any approximation with a single Gaussian function can do no 
better than treating ft,^ as a zero-mean Gaussian random vector 
This is equivalent to treating h'^x'i as Gaussian noise, which 
leads to poor performance. 
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Fig. 9. The BER performance with different number of components in the 
messages. The interference is assumed to be 3 dB weaker than the desired 
signal. The density of pilots is 25%. 



VII. Conclusion 

A novel architecture based on graphical models and belief 
propagation has been proposed for joint channel estimation, 
interference mitigation and decoding. Such joint processing is 
facilitated by efficient iterative message-passing algorithms, 
where the total complexity is essentially the sum of the 
complexity of the components, rather than their product as is 
typical in joint maximum likelihood receivers. In the presence 
of time-varying Rayleigh fading and a strong co-channel 
interference, the message-passing algorithm provides a much 
lower uncoded error floor than linear channel estimation. The 
results with LDPC codes show at least 5 dB gain for achieving 
acceptable bit-error rates. Also, this gain is robust with respect 
to mismatch in channel statistics. 

We have considered only two users with multiple receive 
antennas. Although this is an important case, and the approach 
can be generalized, there may be implementation (complexity) 
issues with extending the algorithm. For example, if we have 
more than one interferer or use larger constellations, the 
number of hypotheses at each message-passing step increases 
significantly. To maintain a target performance, we need to 
increase the number of Gaussian components in each step 
accordingly. Therefore, the complexity may significantly in- 
crease with these extensions. Finally, the algorithm is difficult 
to analyze. While our results give some basic insights into 
performance, relative gains are difficult to predict. 

Directions for future work include extensions to MIMO 
channels (where channel modeling within the message-passing 
framework becomes a challenge) as well as implementation 
issues including methods for reducing complexity. Extending 
the message-passing approach to equalization of frequency 
selective channels with interference is also an interesting 
direction. For example, the narrowband model (1) considered 
here could be viewed as an OFDM system with a number of 
sub-channels. (The receiver algorithm should then be modified 
to account for correlations across sub-channels.) Alternatively, 



message-passing approach could be combined with adaptive 
equalization in the time domain. 
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